Guide to IP Layer Network Administration with Linux

Version 0.4.5

Author: Martin A. Brown

SecurePipe, Inc.
Network Administration

2007-Mar-14

Revision History

Abstract

This guide provides an overview of many of the tools available for IP network administration of the linux operating system, kernels in the 2.2 and 2.4 series. It covers Ethernet, ARP, IP routing, NAT, and other topics central to the management of IP networks.


Table of Contents

Introduction
1. Target Audience, Assumptions, and Recommendations
2. Conventions
3. Bugs and Roadmap
4. Technical Note and Summary of Approach
5. Acknowledgements and Request for Remarks
I. Concepts
1. Basic IP Connectivity
1.1. IP Networking Control Files
1.2. Reading Routes and IP Information
1.2.1. Sending Packets to the Local Network
1.2.2. Sending Packets to Unknown Networks Through the Default Gateway
1.2.3. Static Routes to Networks
1.3. Changing IP Addresses and Routes
1.3.1. Changing the IP on a machine
1.3.2. Setting the Default Route
1.3.3. Adding and removing a static route
1.4. Conclusion
2. Ethernet
2.1. Address Resolution Protocol (ARP)
2.1.1. Overview of Address Resolution Protocol
2.1.2. The ARP cache
2.1.3. ARP Suppression
2.1.4. The ARP Flux Problem
2.2. Proxy ARP
2.3. ARP filtering
2.4. Connecting to an Ethernet 802.1q VLAN
2.5. Link Aggregation and High Availability with Bonding
2.5.1. Link Aggregation
2.5.2. High Availability
3. Bridging
3.1. Concepts of Bridging
3.2. Bridging and Spanning Tree Protocol
3.3. Bridging and Packet Filtering
3.4. Traffic Control with a Bridge
3.5. ebtables
4. IP Routing
4.1. Introduction to Linux Routing
4.2. Routing to Locally Connected Networks
4.3. Sending Packets Through a Gateway
4.4. Operating as a Router
4.5. Route Selection
4.5.1. The Common Case
4.5.2. The Whole Story
4.5.3. Summary
4.6. Source Address Selection
4.7. Routing Cache
4.8. Routing Tables
4.8.1. Routing Table Entries (Routes)
4.8.2. The Local Routing Table
4.8.3. The Main Routing Table
4.9. Routing Policy Database (RPDB)
4.10. ICMP and Routing
4.10.1. MTU, MSS, and ICMP
4.10.2. ICMP Redirects and Routing
5. Network Address Translation (NAT)
5.1. Rationale for and Introduction to NAT
5.2. Application Layer Protocols with Embedded Network Information
5.3. Stateless NAT with iproute2
5.3.1. Stateless NAT Packet Capture and Introduction
5.3.2. Stateless NAT Practicum
5.3.3. Conditional Stateless NAT
5.4. Stateless NAT and Packet Filtering
5.5. Destination NAT with netfilter (DNAT)
5.5.1. Port Address Translation with DNAT
5.6. Port Address Translation (PAT) from Userspace
5.7. Transparent PAT from Userspace
6. Masquerading and Source Network Address Translation
6.1. Concepts of Source NAT
6.1.1. Differences Between SNAT and Masquerading
6.1.2. Double SNAT/Masquerading
6.2. Issues with SNAT/Masquerading and Inbound Traffic
6.3. Where Masquerading and SNAT Break
7. Packet Filtering
7.1. Rationale for and Introduction to Packet Filtering
7.1.1. History of Linux Packet Filter Support
7.2. Limits and Weaknesses of Packet Filtering
7.2.1. Limits of the Usefulness of Packet Filtering
7.2.2. Weaknesses of Packet Filtering
7.2.3. Complex Network Layer Stateless Packet Filters
7.3. General Packet Filter Requirements
7.4. The Netfilter Architecture
7.4.1. Packet Filtering with iptables
7.5. Packet Filtering with ipchains
7.5.1. Packet Mangling with ipchains
7.6. Protecting a Host
7.7. Protecting a Network
7.8. Further Resources
8. Statefulness and Statelessness
8.1.
8.2. Statelessness of IP Routing
8.3. Netfilter Connection Tracking
8.3.1.
8.3.2.
II. Cookbook
9. Advanced IP Management
9.1. Multiple IPs and the ARP Problem
9.2. Multiple IP Networks on one Ethernet Segment
9.3. Breaking a network in two with proxy ARP
9.4. Multiple IPs on an Interface
9.5. Multiple connections to the same Ethernet
9.6. Multihomed Hosts
9.7. Binding to Non-local Addresses
10. Advanced IP Routing
10.1. Introduction to Policy Routing
10.2. Overview of Routing and Packet Filter Interactions
10.3. Using the Routing Policy Database and Multiple Routing Tables
10.3.1. Using Type of Service Policy Routing
10.3.2. Using fwmark for Policy Routing
10.3.3. Policy Routing and NAT
10.4. Multiple Connections to the Internet
10.4.1. Outbound traffic Using Multiple Connections to the Internet
10.4.2. Inbound traffic Using Multiple Connections to the Internet
10.4.3. Using Multiple Connections to the Internet for Inbound and Outbound Connections
11. Scripts for Managing IP
11.1. Proxy ARP Scripts
11.2. NAT Scripts
12. Troubleshooting
12.1. Introduction to Troubleshooting
12.2. Troubleshooting at the Ethernet Layer
12.3. Troubleshooting at the IP Layer
12.4. Handling and Diagnosing Routing Problems
12.5. Identifying Problems with TCP Sessions
12.6. DNS Troubleshooting
III. Appendices and Reference
A. An Example Network and Description
A.1. Example Network Map and General Notes
A.2. Example Network Addressing Charts
B. Ethernet Layer Tools
B.1. arp
B.2. arping
B.3. ip link
B.3.1. Displaying link layer characteristics with ip link show
B.3.2. Changing link layer characteristics with ip link set
B.3.3. Deactivating a device with ip link set
B.3.4. Activating a device with ip link set
B.3.5. Using ip link set to change the MTU
B.3.6. Changing the device name with ip link set
B.3.7. Changing hardware or Ethernet broadcast address with ip link set
B.4. ip neighbor
B.5. mii-tool
C. IP Address Management
C.1. ifconfig
C.1.1. Displaying interface information with ifconfig
C.1.2. Bringing down an interface with ifconfig
C.1.3. Bringing up an interface with ifconfig
C.1.4. Reading ifconfig output
C.1.5. Changing MTU with ifconfig
C.1.6. Changing device flags with ifconfig
C.1.7. General remarks about ifconfig
C.2. ip address
C.2.1. Displaying interface information with ip address show
C.2.2. Using ip address add to configure IP address information
C.2.3. Using ip address del to remove IP addresses from an interface
C.2.4. Removing all IP address information from an interface with ip address flush
C.2.5. Conclusion
D. IP Route Management
D.1. route
D.1.1. Displaying the routing table with route
D.1.2. Reading route's output
D.1.3. Using route to display the routing cache
D.1.4. Creating a static route with route add
D.1.5. Creating a default route with route add default
D.1.6. Removing routes with route del
D.2. ip route
D.2.1. Displaying a routing table with ip route show
D.2.2. Displaying the routing cache with ip route show cache
D.2.3. Using ip route add to populate a routing table
D.2.4. Adding a default route with ip route add default
D.2.5. Setting up NAT with ip route add nat
D.2.6. Removing routes with ip route del
D.2.7. Altering existing routes with ip route change
D.2.8. Programmatically fetching route information with ip route get
D.2.9. Clearing routing tables with ip route flush
D.2.10. ip route flush cache
D.2.11. Summary of the use of ip route
D.3. ip rule
D.3.1. ip rule show
D.3.2. Displaying the RPDB with ip rule show
D.3.3. Adding a rule to the RPDB with ip rule add
D.3.4. ip rule add nat
D.3.5. ip rule del
E. Tunnels and VPNs
E.1. Lightweight encrypted tunnel with CIPE
E.2. GRE tunnels with ip tunnel
E.3. All manner of tunnels with ssh
E.4. IPSec implementation via FreeS/WAN
E.5. IPSec implementation in the kernel
E.6. PPTP
F. Sockets; Servers and Clients
F.1. telnet
F.2. nc
F.3. socat
F.4. tcpclient
F.5. xinetd
F.6. tcpserver
F.7. redir
G. Diagnostic Tools
G.1. ping
G.1.1. Using ping to test reachability
G.1.2. Using ping to stress a network
G.1.3. Recording a network route with ping
G.1.4. Setting the TTL on a ping packet
G.1.5. Setting ToS for a diagnostic ping
G.1.6. Specifying a source address for ping
G.1.7. Summary on the use of ping
G.2. traceroute
G.2.1. Using traceroute
G.2.2. Telling traceroute to use ICMP echo request instead of UDP
G.2.3. Setting ToS with traceroute
G.2.4. Summary on the use of traceroute
G.3. mtr
G.4. netstat
G.4.1. Displaying socket status with netstat
G.4.2. Displaying the main routing table with netstat
G.4.3. Displaying network interface statistics with netstat command
G.4.4. Displaying network stack statistics with netstat
G.4.5. Displaying the masquerading table with netstat
G.5. tcpdump
G.5.1. Using tcpdump to view ARP messages
G.5.2. Using tcpdump to see ICMP unreachable messages
G.5.3. Using tcpdump to watch TCP sessions
G.5.4. Reading and writing tcpdump data
G.5.5. Understanding fragmentation as reported by tcpdump
G.5.6. Other options to the tcpdump command
G.6. tcpflow
G.7. tcpreplay
H. Miscellany
H.1. ipcalc and other IP addressing calculators
H.2. Some general remarks about iproute2 tools
H.3. Brief introduction to sysctl
I. Links to other Resources
I.1. Links to Documentation
I.1.1. Linux Networking Introduction and Overview Material
I.1.2. Linux Security and Network Security
I.1.3. General IP Networking Resources
I.1.4. Masquerading topics
I.1.5. Network Address Translation
I.1.6. iproute2 documentation
I.1.7. Netfilter Resources
I.1.8. ipchains Resources
I.1.9. ipfwadm Resources
I.1.10. General Systems References
I.1.11. Bridging
I.1.12. Traffic Control
I.1.13. IPv4 Multicast
I.1.14. Miscellaneous Linux IP Resources
I.2. Links to Software
I.2.1. Basic Utilities
I.2.2. Virtual Private Networking software
I.2.3. Traffic Control queueing disciplines and command line tools
I.2.4. Interfaces to lower layer tools
I.2.5. Packet sniffing and diagnostic tools
J. GNU Free Documentation License
J.1. PREAMBLE
J.2. APPLICABILITY AND DEFINITIONS
J.3. VERBATIM COPYING
J.4. COPYING IN QUANTITY
J.5. MODIFICATIONS
J.6. COMBINING DOCUMENTS
J.7. COLLECTIONS OF DOCUMENTS
J.8. AGGREGATION WITH INDEPENDENT WORKS
J.9. TRANSLATION
J.10. TERMINATION
J.11. FUTURE REVISIONS OF THIS LICENSE
J.12. ADDENDUM: How to use this License for your documents
Reference Bibliography and Recommended Reading
Index

List of Tables

2.1. Active ARP cache entry states
4.1. Keys used for hash table lookups during route selection
5.1. Filtering an iproute2 NAT packet with ipchains
A.1. Example Network; Network Addressing
A.2. Example Network; Host Addressing
B.1. ip link link layer device states
B.2. Ethernet Port Speed Abbreviations
C.1. Interface Flags
C.2. IP Scope under ip address
G.1. Possible Session States in netstat output
H.1. iproute2 Synonyms

List of Examples

1.1. Sample ifconfig output
1.2. Testing reachability of a locally connected host with ping
1.3. Testing reachability of non-local hosts
1.4. Sample routing table with a static route
1.5. ifconfig and route output before the change
1.6. Bringing down a network interface with ifconfig
1.7. Bringing up an Ethernet interface with ifconfig
1.8. Adding a default route with route
1.9. Adding a static route with route
1.10. Removing a static network route and adding a static host route
2.1. ARP conversation captured with tcpdump
2.2. Gratuitous ARP reply frames
2.3. Unsolicited ARP request frames
2.4. Duplicate Address Detection with ARP
2.5. ARP cache listings with arp and ip neighbor
2.6. ARP cache timeout
2.7. ARP flux
2.8. Correction of ARP flux with conf/$DEV/arp_filter
2.9. Correction of ARP flux with net/$DEV/hidden
2.10. Proxy ARP Network Diagram
2.11. Bringing up a VLAN interface
2.12. Link aggregation bonding
2.13. High availability bonding
4.1. Classes of IP addresses
4.2. Using ipcalc to display IP information
4.3. Identifying the locally connected networks with route
4.4. Routing Selection Algorithm in Pseudo-code
4.5. Listing the Routing Policy Database (RPDB)
4.6. Typical content of /etc/iproute2/rt_tables
4.7. unicast route types
4.8. broadcast route types
4.9. local route types
4.10. nat route types
4.11. unreachable route types
4.12. prohibit route types
4.13. blackhole route types
4.14. throw route types
4.15. Kernel maintenance of the local routing table
4.16. unicast rule type
4.17. nat rule type
4.18. unreachable rule type
4.19. prohibit rule type
4.20. blackhole rule type
4.21. ICMP Redirect on the Wire
5.1. Stateless NAT Packet Capture
5.2. Basic commands to create a stateless NAT
5.3. Conditional Stateless NAT (not performing NAT for a specified destination network)
5.4. Using an ipchains packet filter with stateless NAT
5.5. Using DNAT for all protocols (and ports) on one IP
5.6. Using DNAT for a single port
5.7. Simulating full NAT with SNAT and DNAT
7.1. Blocking a destination and using the REJECT target, cf. Example D.17, “Adding a prohibit route with route add
10.1. Multiple Outbound Internet links, part I; ip route
10.2. Multiple Outbound Internet links, part II; iptables
10.3. Multiple Outbound Internet links, part III; ip rule
10.4. Multiple Internet links, inbound traffic; using iproute2 only
11.1. Proxy ARP SysV initialization script
11.2. Proxy ARP configuration file
11.3. Static NAT SysV initialization script
11.4. Static NAT configuration file
B.1. Displaying the arp table with arp
B.2. Adding arp table entries with arp
B.3. Deleting arp table entries with arp
B.4. Displaying reachability of an IP on the local Ethernet with arping
B.5. Duplicate Address Detection with arping
B.6. Using ip link show
B.7. Using ip link set to change device flags
B.8. Deactivating a link layer device with ip link set
B.9. Activating a link layer device with ip link set
B.10. Using ip link set to change device flags
B.11. Changing the device name with ip link set
B.12. Changing broadcast and hardware addresses with ip link set
B.13. Displaying the ARP cache with ip neighbor show
B.14. Displaying the ARP cache on an interface with ip neighbor show
B.15. Displaying the ARP cache for a particular network with ip neighbor show
B.16. Entering a permanent entry into the ARP cache with ip neighbor add
B.17. Entering a proxy ARP entry with ip neighbor add proxy
B.18. Altering an entry in the ARP cache with ip neighbor change
B.19. Removing an entry from the ARP cache with ip neighbor del
B.20. Removing learned entries from the ARP cache with ip neighbor flush
B.21. Detecting link layer status with mii-tool
B.22. Specifying Ethernet port speeds with mii-tool --advertise
B.23. Forcing Ethernet port speed with mii-tool --force
C.1. Viewing interface information with ifconfig
C.2. Bringing down an interface with ifconfig
C.3. Bringing up an interface with ifconfig
C.4. Changing MTU with ifconfig
C.5. Setting interface flags with ifconfig
C.6. Displaying IP information with ip address
C.7. Adding IP addresses to an interface with ip address
C.8. Removing IP addresses from interfaces with ip address
C.9. Removing all IPs on an interface with ip address flush
D.1. Viewing a simple routing table with route
D.2. Viewing a complex routing table with route
D.3. Viewing the routing cache with route
D.4. Adding a static route to a network route add
D.5. Adding a static route to a host with route add
D.6. Adding a static route to a host on the same media with route add
D.7. Setting the default route with route
D.8. An alternate method of setting the default route with route
D.9. Removing a static host route with route del
D.10. Removing the default route with route del
D.11. Viewing the main routing table with ip route show
D.12. Viewing the local routing table with ip route show table local
D.13. Viewing a routing table with ip route show table
D.14. Displaying the routing cache with ip route show cache
D.15. Displaying statistics from the routing cache with ip -s route show cache
D.16. Adding a static route to a network with route add, cf. Example D.4, “Adding a static route to a network route add
D.17. Adding a prohibit route with route add
D.18. Using from in a routing command with route add
D.19. Using src in a routing command with route add
D.20. Setting the default route with ip route add default
D.21. Creating a NAT route for a single IP with ip route add nat
D.22. Creating a NAT route for an entire network with ip route add nat
D.23. Removing routes with ip route del
D.24. Altering existing routes with ip route change
D.25. Testing routing tables with ip route get
D.26. Removing a specific route and emptying a routing table with ip route flush
D.27. Emptying the routing cache with ip route flush cache
D.28. Displaying the RPDB with ip rule show
D.29. Creating a simple entry in the RPDB with ip rule add
D.30. Creating a complex entry in the RPDB with ip rule add
D.31. Creating a NAT rule with ip rule add nat
D.32. Creating a NAT rule for an entire network with ip rule add nat
D.33. Removing a NAT rule for an entire network with ip rule del nat
F.1. Simple use of nc
F.2. Specifying timeout with nc
F.3. Specifying source address with nc
F.4. Using nc as a server
F.5. Delaying a stream with nc
F.6. Using nc with UDP
F.7. Simple use of socat
F.8. Using socat with proxy connect
F.9. Using socat perform SSL
F.10. Connecting one end of socat to a file descriptor
F.11. Connecting socat to a serial line
F.12. Using a PTY with socat
F.13. Executing a command with socat
F.14. Connecting one socat to another one
F.15. Simple use of tcpclient
F.16. Specifying the local port which tcpclient should request
F.17. Specifying the local IP to which tcpclient should bind
F.18. IP redirection with xinetd
F.19. Publishing a service with xinetd
F.20. Simple use of tcpserver
F.21. Specifying a CDB for tcpserver
F.22. Limiting the number of concurrently accept TCP sessions under tcpserver
F.23. Specifying a UID for tcpserver's spawned processes
F.24. Redirecting a TCP port with redir
F.25. Running redir in transparent mode
F.26. Running redir from another TCP server
F.27. Specifying a source address for redir's client side
G.1. Using ping to test reachability
G.2. Using ping to specify number of packets to send
G.3. Using ping to specify number of packets to send
G.4. Using ping to stress a network
G.5. Using ping to stress a network with large packets
G.6. Recording a network route with ping
G.7. Setting the TTL on a ping packet
G.8. Setting ToS for a diagnostic ping
G.9. Specifying a source address for ping
G.10. Simple usage of traceroute
G.11. Displaying IP socket status with netstat
G.12. Displaying IP socket status details with netstat
G.13. Displaying the main routing table with netstat
G.14. Displaying the routing cache with netstat
G.15. Displaying the masquerading table with netstat
G.16. Viewing an ARP broadcast request and reply with tcpdump
G.17. Viewing a gratuitous ARP packet with tcpdump
G.18. Viewing unicast ARP packets with tcpdump
G.19. tcpdump reporting port unreachable
G.20. tcpdump reporting host unreachable
G.21. tcpdump reporting net unreachable
G.22. Monitoring TCP window sizes with tcpdump
G.23. Examining TCP flags with tcpdump
G.24. Examining TCP acknowledgement numbers with tcpdump
G.25. Writing tcpdump data to a file
G.26. Reading tcpdump data from a file
G.27. Causing tcpdump to use a line buffer
G.28. Understanding fragmentation as reported by tcpdump
G.29. Specifying interface with tcpdump
G.30. Timestamp related options to tcpdump